Nearest Neighbor Classification

The nearest-neighbor method is perhaps the simplest of all algorithms for predicting the class of a test example. The training phase is trivial: simply store every training example, with its label. To make a prediction for a test example, first compute its distance to every training example. Then, keep the k closest training examples, where k ≥ 1 is a fixed integer. Look for the label that is most common among these examples. This label is the prediction for this test example. This basic method is called the kNN algorithm. There are two design choices to make: the value of k, and the distance function to use. When there are two alternative classes, the most common choice for k is a small odd integer, for example k = 3. When each example is a fixed-length vector of real numbers, the most common distance function is Euclidean distance: